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ABSTRACT 

The procedures for the selection of a test 
population to use and evaluate the short units developed by the 
Sociological Resources for the Social Studies Project (SRSS), 
"Episodes in Social Inquiry Series," are considered. They wanted to 
assess the effectiveness of the materials under a variety of 
circumstances; classes from different parts of the country, from 
communities of various sizes and economic strata, with students of 
several ability levels, in a variety of social studies courses, in 
botii large and small schools. Some of the problems discussed are; 1) 
school selection by geographical location; 2) working with the 
authority structure in public school systems; 3) deciding which 
contact in a school would be most receptive; 4) selecting the 
participant classes within a school; i) soliciting the cooperation of 
the teachers; 6) establishing clear communications with the teachers 
about their role, the experimental nature of the episodes, and the 
desire for honest and candid feedback. The project concluded; when 
formative evaluation is the goal and time is an important factor, a 
test population selected on the basis of wlu will complete evaluation 
tasks is more important than a representative sampling of the total 
universe of potential users* ED 03S 563 and SO 000 246 are related 
documents. (SBE) 
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SELECTING A TEST POPULATION FOR EVALUATING . 

THE SOCIOLOGICAL RESOURCES PROJECT MATERIALS 1 

2 

F. Lincoln Grahlfs, Brookdalo Community College 
W. M. Hering, Jr., Sociological Resources for the Social Studies 

Sociological Resources for the Social Studies is a project of the American 
Sociological Association, supported by the National Science Foundation. In 
existence since 196h, 8RSS was mandated to produce three sorts of materials for 
the secondary school social studies curriculum: a series of paperback books 

containing significant sociological articles rewritten for high school students 
and topically compiled, a one semester sociology course, and a series of some 
forty episodes, each dealing with a sociological topic, designed to be inte- 
grated into existing social studies courses, and incorporating an Inquiry 
approach to teaching and learning. Tills paper will consider the selection of 
a test population for the episodes} the development of these tvo-veek units has 
constituted the main activity of SRSS. The evaluation is in its final stages 
with the last six episodes being field tested this semester. It began in 1966} 
since that initial effort procedures and techniques have undergone many changes. 

We made an initial decision to test the episodes in pairs, using the stu- 
dents studying one episode as control for the other. This was accomplished by 
administering the forty-item multiple-choice test covering one episode as a pre- 
test t,o the 1500 students who were to study its counterpart. This procedure is 
•till being used. 



This paper is one part of a symposium on evaluation of social studies curric- 
ulum projects presented at the annual meeting of the American Educational 
Research Association, Minneapolis, March 2-6, 1970. 



® From 1966 to 1969 Mr. Grahlfs was Supervisor of Evaluation, Sociological 
Resources for the Social Studies. 
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One of the first steps in the evaluation was the selection of a test popu- 
lation, a term which was substituted for sample because ve had made certain 
arbitrary decisions regarding its composition rather than making a totally ran- 
dom selection. We wanted to assess the effectiveness of our materials under a 
variety of circumstances, so wo included classes from different parts of the 
country, from communities of various sites and representative of ft variety of 
economic strata. We were also interested in obtaining students of several 
ubility levels, in a variety of social studies courses, in both large and snail 
schools. 

The project had, at the time, been in existence for two years and ve had an 
ample file of names of teachers who had written expressing interest in the pro- 
ject and the materials it was developing. We rejected, at this point, a sugges- 
tion to draw our test population from such a file, on the assumption that is 
consisted of teachers who might be categorised ob atypical of ell potential users 

of the material. We decided instead to have the test, population drawn for us by 
3 

a contractor, according to specifications which we would determine. 

Our first consideration was geographic distribution. It was decided to use 
a "cluster" approach, drawing an equal number of schools from each of four major 
census regions in the United StAtes. Primarily for convenience in visiting the 
schools involved, we focused our test populations around Hew York City on the 
East Coast, i,os Angeles on the Vest Coast, New Orleans in the South and, in the 
Midwest, around Indianapolis and Pes Moines. High refusal rate, coupled with 
time pressure caused by the early start of summer vacations in that region led 
us to augment the southern cluster by including sene schools in Tennessee. In 

^ Ultimately, the contractor chosen for this and for testing and statistical 
work, was The Psychological Corporation. 
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our instruct ions to the contractor ve specified a 3trntified random selection of 
schools in the designated areas • One of the two dimensions on which they were to 
stratify was communities classified into "metropolitan," "suburban," and "all 
other." The second dimension var the size of the student body. 

Having resolved the question of school selection, ve faced the dilemma of 
deciding which contact in the school would be moat receptive. Our initial request 
was made to the high school principal. In retrospect, this secir.s to have been a 
reasonably good choice. In most cases the principal passed the request along to 
his social studies chairman who contacted us; in some instances he sent us the 
name of his superior through whom ve would have to operate. Vo did lose some of 
the selected schools through this procedure, either because the principal was not 
much interested and didn't pass the letter on or because the assistant superin- 
tendent or coordinator felt we had bypassed him and vetoed the entire natter. 

Ve did get a variety of classroom situations. Ve also learned a few things 
about the structure of authority in public schools. 

Our planning for the I 9 CV-O 8 series of trials began early, and va6 both 
elaborate and ambitious; ve wanted to improve upon our previous performance. 
SpeaXing in retrospect, we merely created more work for ourselves. Ve hired a 
specialist in sampling who made a random selection of ten staten t* ratified by 
geographic region and by per capita expenditure cn education. Ve then obtained 
from the National Association of Science Teachers, a list of all secondary 
schools in those states, indicating the number of social studies teachers in each. 
From this our specialist provided us with a sample of schools. When ve had 
obtained consent of the necessary personnel (this tine ve started with the super- 
intendent) ve sent a for* to the social studies cheinaan, asking him to list all 
teachers In hi# department and the classes they taught. Using a table of random 
numbers , we selected classes and solicited cooperation from, the teachers. 
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’ Our purpose was to eliminate the bias of self-selection; it didn't vork. 

We encountered on excessive refusal rate and many substitutions vore made, not 
only by us, but by the people in the schools. V'e did, however, nrofit by pre- 
vious experience in one way. As soon as the states hod been selected we sent a 
letter to the top educational administrator and the state social studies coor- 
dinator in each, informing then of our nlans; when the schools had been selected 
ve wrote first to the superintendent of schools; after a reasonable tiro, if we 
had no response ve wrote to the urinciral . We touched all bases in the "c^rin 
of comand." This proved to be a vise policy. 

At the end of this second year ve again reviewed our experience and con- 
cluded that the ''cluster" approach of the first year was much more usable, and 
that we should return to it. Ve could, we felt, improve upon the cluster approach 
by having a single person throuph vhoo we would communicate in each area. 

Ve selected areas which ve felt gave reasonable geographic diversity and 
relied on our professional contacts to establish harmonious liaison with the 
social studies supervisor in each. Tuese supervisors would help us select schools 
reflecting the desired diversity of social and economic characteristics and would 
advise us concerning rural schools reasonably close to the city and suburbs. 

Under this plan, we obtained the cooperation cf social studies coordinators 
from Seattle; Minneapolis; New Orleans; Cleveland; Springfield, Massachusetts; and 
Hartford, Connecticut. More importantly, perhaps* ve obtained consent of their 
school systems for these people to act as our agents in enlisting participating 
teachers. These people vere then brought to 3RSS headquarters in Ann Arbor for a 
thorough one-day briefing on materials and procedures. 

This has clearly been a most satisfactory approach. We obtained the variety 
Of claseroctts ve desired, ve had a clearer lihe or ccraunic&tion, and people close 
to the "firing line” who could both coordinate and trouble shoot. 



5 



There vere the usual (and gome unusal) problems. Come episodes had more 
appealing titles than others and we could not always /ret the full complement 
of fifty classes for each. The supervisors vere reticent to use a heavy hand 
in suggesting that teachers accept an episode which they had not originally 
requested; vo did not want to encourage them to do this, not only because ve 
thought it might create a negative bias, but because ve felt teachers should 
not be comoelled to teach material not in agreement with their goals. In one 
city the AFT contract made teacher participation meager, and ve had to seek 
more teachers in the other clusters. Although ve agreed to pay teachers a 
small honorarium for completing our lengthy questionnaire, some kept the 
materials in lieu of honorarium. Some teachers failed to understand their 
role, and u.^cd the episodes in situations of little use to us. No matter hov 
hard ve tried, one of our most difficult problems was the establishment of 
clear comnuncations with the teachers about their rcle, the experimental 
nature of the episodes, and our desire for honest and candid feedback. The 
pre and posttest situation v a s especially confusing; many students and teachers 
simply could not understand vhy ve administered a test over material which had 
not been studied, and which would not be studied. Because ve knew that the 
final, published product would not be distributed with personal advice about 
its use, ve avoided pre-evaluation visits to the teachers, hoping to insure a 
realistic trial. Despite these problems ve vere able to select vhat seemed to 

be a test population which provided us with the necessary formative feedback 
for revision. 

This year ve have made only one major change in the selection process. Two 
systems were especially successful in enlisting a large number of teachers and 
ve asked then to continue their affiliation vith us another year. To these ve 
added three areas not previously used: Atlanta, Miami , an d the San Francisco Bay 

Area. In each of these areas the National Pcience Foundation vas supporting 
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in-service institutes in sociology and ve decided to include participants in 
those institutes in our evaluation. Because the institutes included on their 
staffs supervisors from the school systems, the selection of a test population 
vas iccomplished in much the some manner as before. Some teachers in these 
areas vere not enrolled in the institutes and we encouraged them to evaluate 
episodes so that ve could get a variety of teacher backgrounds. 

It is too soon to tell if this use of in-service institutes has made much 
difference in the quality of the evaluation \ it has made the selection of a 
test population much easier. Because teachers can look over an episode together, 
and discuss it during the institute meetings, ve have come closer to ouf goal of 
fifty classes for each episode. 

Our conclusion will not surprise those who have been involved in similar 
efforts, but it may offend eome in the research ooumunity. Vhen formative evalu- 
ation is the goal, and vhen time is an important factor, a test population selected 
on the basis of likelihood that they will complete evaluation tasks far outweighs 
in advantage a more representative sampling of the total universe of potential 
users of the materials. 
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